Evaluating Speech Separation Systems

Common evaluation standards are critical to making progress in any field, but they can also distort research by shifting all the attention to a limited subset of the problem. Here, we consider the problem of evaluating algorithms for speech separation and acoustic scene analysis, noting some weaknesses of existing measures, and making some suggestions for future evaluations. We take the position that the most relevant ‘ground truth’ for sound mixture organization is the set of sources perceived by human listeners, and that best evaluation standards would measure the machine’s match to this perception at a level abstracted away from the low-level signal features most often considered in signal processing.